Skip to content

deps: update logparser with structured log level detection#235

Merged
mayankpande88 merged 1 commit into
mainfrom
deps/logparser-structured-level-detection
Apr 6, 2026
Merged

deps: update logparser with structured log level detection#235
mayankpande88 merged 1 commit into
mainfrom
deps/logparser-structured-level-detection

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

What changed in logparser

ParseStructuredLog() extracts level from JSON "level"/"severity"/"levelname"/"log_type" fields instead of scanning raw text for keywords. This fixes:

  • INFO log with "type":"ErrorRate" misclassified as ERROR
  • INFO log with "msg":"error processing request" misclassified as ERROR
  • Success log with "data":{"observation":"fatal: ..."} misclassified as CRITICAL

Performance: 18% faster full pipeline (eliminates redundant JSON parse), 8% fewer allocations.

Test plan

Updates github.com/nudgebee/logparser to include single-pass
structured log parsing that extracts level from JSON/logfmt fields
instead of unreliable text scanning (nudgebee/logparser#18).

Fixes false positive log level classification where INFO logs
containing "error"/"fatal" in message content were misclassified
as ERROR/CRITICAL.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the github.com/nudgebee/logparser dependency to a newer version. Feedback was provided regarding the presence of stale entries in the go.sum file, suggesting that go mod tidy should be executed to maintain a clean dependency list.

Comment thread go.sum
@mayankpande88 mayankpande88 merged commit 981e9d3 into main Apr 6, 2026
1 check passed
@mayankpande88 mayankpande88 deleted the deps/logparser-structured-level-detection branch April 6, 2026 06:14
mayankpande88 added a commit that referenced this pull request Apr 9, 2026
* deps: update logparser with structured log level detection (#235)

Updates github.com/nudgebee/logparser to include single-pass
structured log parsing that extracts level from JSON/logfmt fields
instead of unreliable text scanning (nudgebee/logparser#18).

Fixes false positive log level classification where INFO logs
containing "error"/"fatal" in message content were misclassified
as ERROR/CRITICAL.

* chore(deps): bump google.golang.org/grpc from 1.76.0 to 1.79.3 (#229)

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.76.0 to 1.79.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.76.0...v1.79.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.79.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: prevent OOM crash from L7 protocol misidentification (#236)

The ClickHouse eBPF detector matched non-ClickHouse TCP payloads using
a 3-byte heuristic (0x01, 0x00, 0x01), causing ch-go to decode garbage
varint lengths and attempt 228TB allocations — a fatal OOM that bypasses
defer/recover.

3-layer defense:

Layer 0 — eBPF detection hardening:
  Remove non-port-gated is_clickhouse_query() call. ClickHouse native
  protocol detection now only on ports 9000/8123.

Layer 1 — Bounded parsing:
  Wrap ch-go proto.NewReader with io.LimitReader(payload size). Varint
  lengths exceeding actual payload now return io.EOF instead of OOM.

Layer 2 — Userspace validation:
  Add structural checks before invoking external libraries:
  - ClickHouse: validate query code byte + query ID length
  - Postgres: reject unknown frame types (Q/B/P/C only)
  - Zookeeper: validate opcode in known range

Layer 3 — Protocol reclassification:
  Track consecutive parse failures per connection. After 3 failures,
  override the cached protocol to stop further misidentified parsing.
  Logs a warning for observability.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
mayankpande88 added a commit that referenced this pull request Apr 9, 2026
* fix: update logparser to v0.0.0-20260310062405-9d6258bdd771

Picks up fix/json-log-pattern-hashing (PR #16) which stabilizes
JSON log pattern hashing by extracting only message fields.

* fix: configurable sensitive log detection with optimized logparser (#224)

* fix: update logparser and add configurable sensitive log detection flags

- Update logparser to v0.0.0-20260313095946-b22168b95d9c with optimized
  sensitive parsing (anchor pre-filter, confidence tiers, singleton cache)
- Add SENSITIVE_LOG_SAMPLE_RATE, SENSITIVE_LOG_MIN_CONFIDENCE,
  SENSITIVE_LOG_MAX_DETECTIONS flags for fine-grained control
- Update all NewParser callsites to use SensitiveConfig struct
- Remove local replace directive

* fix: deduplicate SensitiveConfig and align flag naming

- Extract SensitiveConfig into a single variable reused across all 3 NewParser calls
- Rename flag/env var to sensitive-log-max-detections-per-container / SENSITIVE_LOG_MAX_DETECTIONS_PER_CONTAINER to match variable name

* fix: cap HTTP/2 DATA payload accumulation to prevent OOM (#226)

Http2Parser.Parse appends DATA frame payloads to RequestPayload and
ResponsePayload with no size limit. On nodes with high-throughput
HTTP/2 traffic (envoy-gateway, gRPC proxies, benchmark-server), these
grow unbounded — confirmed via pprof: 149MB (62% of heap) consumed by
Http2Parser on a pod that OOMs after ~22h.

Cap both payloads at 128KB per stream. They are only used for LLM
provider detection and trace spans which need at most a few KB.
END_STREAM tracking is unaffected so stream completion still works.

* fix: graceful shutdown, L7 panic protection, and bounded strippedGoExeCache (#225)

- Replace os.Exit(0) in signal handler with context cancellation so deferred
  cleanup (cr.Close, profiling.Stop, resolver.StopWatching) runs on shutdown.
  Previously leaked eBPF programs, uprobes, and perf buffers on every restart.
  Use http.Server.Shutdown for graceful HTTP server termination. (fixes #207)

- Wrap L7 event processing with recover() to prevent a single malformed packet
  from crashing the entire agent. Covers both processL7Event and pending L7
  retry paths. (fixes #210)

- Replace unbounded sync.Map with LRU cache (cap 1000) for strippedGoExeCache
  to prevent slow memory growth on CI/CD nodes with many unique Go binaries.
  hashicorp/golang-lru is already a dependency. (fixes #214)

* fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes (#227)

* fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes

HTTP/2 frame parsing was consuming 70% of CPU (profiled). Most HTTP/2
traffic (gRPC, K8s API, service mesh) doesn't need payload capture —
only status codes for L7 metrics.

- Add lightweight mode to Http2Parser: skips DATA payload accumulation
  and RequestHeaders map allocation for non-LLM traffic
- Auto-upgrade to full mode when :authority matches an LLM provider
- Reuse statuses/grpcStatuses maps across Parse() calls (clear instead
  of alloc) — eliminates ~15% CPU from per-call map allocation
- Cap activeRequests at 100 per connection to prevent orphan stream growth
- Cap HTTP/2 parsers at 50 per container
- Fix parser GC leak: connectionless parsers now cleaned up when pid dies
- Eliminate duplicate GetActiveStreamsForLLM() call per event
- Skip LLM stream tracker entirely when parser is lightweight

* fix: address PR review — GC leak for closed connections, deduplicate END_STREAM

- Fix parser GC: closed connection parsers in long-running processes were
  not cleaned up. Now uses heuristic: if pid is alive but no connection
  exists, clean up parsers with zero active requests and no partial data.
- Add HasPartialData() to Http2Parser for safe GC decisions.
- Deduplicate END_STREAM handling: extract it before lightweight/full branch
  instead of duplicating in both paths.

* fix: skip-set for non-LLM HTTP/2 traffic to eliminate CPU waste

processHTTP2WithoutConnection produces zero L7 metrics (no connection =
no DestinationKey). Its only purpose is LLM detection, yet it was parsing
every HTTP/2 frame from gRPC/K8s API traffic — consuming 70% of CPU on
busy nodes.

Two-phase skip-set approach:
1. If dstIP is known, check DNS cache — skip immediately if not LLM.
2. Otherwise, parse first event per pid:fd in lightweight mode to extract
   :authority. If not LLM, add to skip-set and never parse that fd again.

Also adds GC for skip-set entries when the pid dies.

* fix: prevent runtime crash from uint64-to-int overflow in event readers (#228)

* fix: prevent runtime crash from uint64-to-int overflow in event readers

Both perf and ring buffer L7 event readers convert eBPF-reported payload
sizes from uint64 to int without safe clamping. If eBPF sends corrupted
data with high-bit-set sizes (e.g. 0x8000000000000001), int() produces
a negative value that bypasses the MaxPayloadSize cap, causing
make([]byte, negative) to crash the reader goroutine.

Also upgrades cilium/ebpf v0.20.0 → v0.21.0 which includes ring buffer
and memory management fixes relevant to Go 1.24.

Fixes runtime crashes: "fatal error: stack not a power of 2" observed on
12/19 pods after ~7 hours of operation.

* fix: revert cilium/ebpf upgrade, keep clampSize fix only

cilium/ebpf v0.21.0 has breaking API changes (removed Sym, RewriteConstants,
MapName) that affect transitive dependencies. Revert to v0.20.0 and keep
just the uint64-to-int overflow fix.

* fix: evict stale CounterVec label combinations to prevent series explosion (#231)

CounterVec entries for closed TCP connections were never removed,
causing series count to grow linearly with pod uptime. On a 40h pod,
89% of emitted counter series were stale (85/95 destinations in a
single container had no active connection). At customer scale this
accumulated to 989K series (54% of all Prometheus series).

Track label combinations pushed to each CounterVec and periodically
delete entries whose destinations are no longer in activeConnections.
Eviction runs every gcInterval (5min) from the event handler goroutine,
taking mu.Lock to exclude concurrent collect() calls.

* chore: main to test (#237)

* deps: update logparser with structured log level detection (#235)

Updates github.com/nudgebee/logparser to include single-pass
structured log parsing that extracts level from JSON/logfmt fields
instead of unreliable text scanning (nudgebee/logparser#18).

Fixes false positive log level classification where INFO logs
containing "error"/"fatal" in message content were misclassified
as ERROR/CRITICAL.

* chore(deps): bump google.golang.org/grpc from 1.76.0 to 1.79.3 (#229)

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.76.0 to 1.79.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.76.0...v1.79.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.79.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: prevent OOM crash from L7 protocol misidentification (#236)

The ClickHouse eBPF detector matched non-ClickHouse TCP payloads using
a 3-byte heuristic (0x01, 0x00, 0x01), causing ch-go to decode garbage
varint lengths and attempt 228TB allocations — a fatal OOM that bypasses
defer/recover.

3-layer defense:

Layer 0 — eBPF detection hardening:
  Remove non-port-gated is_clickhouse_query() call. ClickHouse native
  protocol detection now only on ports 9000/8123.

Layer 1 — Bounded parsing:
  Wrap ch-go proto.NewReader with io.LimitReader(payload size). Varint
  lengths exceeding actual payload now return io.EOF instead of OOM.

Layer 2 — Userspace validation:
  Add structural checks before invoking external libraries:
  - ClickHouse: validate query code byte + query ID length
  - Postgres: reject unknown frame types (Q/B/P/C only)
  - Zookeeper: validate opcode in known range

Layer 3 — Protocol reclassification:
  Track consecutive parse failures per connection. After 3 failures,
  override the cached protocol to stop further misidentified parsing.
  Logs a warning for observability.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants